As the voids and pits image of bridge often has uneven illumination and multi-background interference problems, an image segmentation algorithm for complex voids and pits of bridge was proposed based on HSI color space and gray fluctuation. Firstly, S-component gray curve was plotted and all the potential peaks and troughs of the curve were searched, then the height differences between adjacent peaks and thoughs were calculated. Secondly, partial height differences were selected based on the standard deviation of gray pixel difference value. Finally the threshold segmentation of image was finished by searching the best threshold based on the standard deviation of partial height differences. Experimental results show that the proposed algorithm has better segmentation effect and real-time performance than OTSU, Niblack and Tsallis entropy method.